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ABSTRACT 



The 1998 NAEP (National Assessment of Educational Progress) 



state level reading assessment measured fourth- and eighth-grade students' 
reading proficiency in situations that involved reading for three purposes: 
reading for literary experience, reading to gain information, and reading to 
perform a task (grade 8 only) . This report describes the reading proficiency 
of Montana fourth- and eighth-grade public school students and eighth-grade 
nonpublic school students and compares their overall performance to students 
in the West region of the United States and the nation (using data from the 
NAEP national assessments) . The distribution of reading results and reading 
achievement level results are provided for subpopulations of students 
(gender, race/ethnicity, students' reports of parents' highest educational 
level, free/reduced-price lunch program eligibility, and type of location) . 
The average reading proficiency of fourth-grade public school students in 



Montana on the NAEP reading scale was 226 compared to 215 nationwide; 287 
compared to 281 nationwide for eight-grade nonpublic school students; and 270 
compared to 261 nationwide for eighth-grade public school students. (RS) 
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What is The Nation's Report Card? 

THE NATION’S REPORT CARD, the National Assessment of Educational Progress (NAEP), is the only nationally 
representative and continuing assessment of what America’s students know and can do in various subject areas. Since 1969, 
assessments have been conducted periodically in reading, mathematics, science, writing, history, geography, and other 
fields. By making objective information on student performance available to policymakers at the national, state, and local 
levels, NAEP is an integral part of our nation’s evaluation of the condition and progress of education. Only information 
related to academic achievement is collected under this program. NAEP guarantees the privacy of individual students and 
their families. 

NAEP is a congressionallv mandated project of the National Center for Education Statistics, the U.S. Department of 
Education. The Commissioner of Education Statistics is responsible, by law, for carrying out the NAEP project through 
competitive awards to qualified organizations. NAEP reports directly to the Commissioner, who is also responsible 
for providing continuing reviews, including validation .studies and solicitation of public comment, on NAEP’s conduct 
and usefulness. * - - 

In 1988, Congress established the National Assessment Governing Board (NAGB) to formulate policy guidelines for 
NAEP. The Board is responsible for selecting the subject areas to be assessed from among those included in the National 
Education Goals; for setting appropriate student performance levels; for developing assessment objectives and test 
specifications through a national consensus approach; for designing the assessment methodology; for developing guidelines 
for reporting and disseminating NAEP results; for developing standards and procedures for interstate, regional, and national 
comparisons; for determining the appropriateness of test items and ensuring they are free from bias; and for taking actions 
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Overview of the NAEP Reading 



What Is NAEP? 

The National Assessment of Educational Progress (NAEP) is the only nationally 
representative and continuing assessment of what students in the United States know and 
can do in various academic subjects. NAEP is authorized by Congress and directed by 
the National Center for Education Statistics (NCES). The National Assessment 
Governing Board (NAGB), an independent body, provides policy guidance for NAEP. 
The NAEP assessments are administered to representative samples of students at the 
national level as well as at the state level for those states that want to participate. To 
ensure comparability in a particular subject across all jurisdictions, NCES has 
established guidelines for school and student participation rates. These guidelines, as 
well as other technical aspects of the assessment, are detailed in the NAEP 1998 Reading 
Report Card ' 1 

The 1998 NAEP program included state-level assessments in reading at grades 4 
and 8 and in writing at grade 8, and national-level assessments in civics, reading, and 
writing at grades 4, 8, and 12. This report and its companion, the NAEP 1998 Reading 
Report Card , provide a first look at the results of the NAEP 1998 reading assessment. 
The Reading Report Card offers additional state-level data. Each participating 
jurisdiction receives its own customized State Report identical in format to this one. 
Summary data tables providing information for all jurisdictions for which results are 
reported in 1998 are available at http://nces.ed.gov/naep/, the NAEP Web site. 

What Is Reported Here? 

The NAEP reading assessment has been administered at the state level three times: in 
public schools at grade 4 in 1992, in public and nonpublic schools at grade 4 in 1994, 
and in public and nonpublic schools at grades 4 and 8 in 1998. Montana participated 
in the 1998 NAEP reading assessment at grades 4 and 8. Montana also participated in 
the NAEP reading assessment in 1994. Both public and nonpublic schools participated. 
However, minimum participation guidelines for publication were not met for grade 4 
nonpublic schools in 1994 or 1998. Montana’s public school results for 1994 and 1998 
and grade 8 nonpublic school results for 1998 are presented here, along with national 
and regional results for comparison. 

This report has two sections. This Overview provides basic information on NAEP 
and the overall results for public schools in graphic form. It describes the assessment, 
the sample of students assessed, the metrics for reporting student performance, and how 
the differences in performance are reported. The second section, Overall Reading 
Performance and Performance by Demographic Characteristics, reports findings for 
the entire public school population at grades 4 and 8 as well as for the population broken 
out by major demographic categories and school type. This information is presented in 
data tables. 

1 Donahue, P. L., Voelkl, K. E., Campbell, J. R., and Mazzeo, J. (1999). The NAEP 1998 reading report card for the 
nation and the states (NCES Publication No. 1999-500). Washington, DC: National Center for Education Statistics. 
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In addition, this report has two appendices. Appendix A, Where to Find More 
Information, describes the data available on the Web and provides information on 
sources of related data. Appendix B, Figures from Section 1, displays full-page 
replicas of Figures 1-5. 

The demographic data provided in this report are only a small portion of the data 
available from the several hundred questions asked of students, teachers, and school 
principals in order to provide context for NAEP results. Overall results for all student 
and school variables for public school students in each participating jurisdiction are 
available in summary data tables at the NAEP Web site. 

How Are Results Reported? 

In this report, as in other NAEP reports, only those results based on preestablished 
minimum sample sizes are reported. For details, see the forthcoming NAEP 1998 
Technical Report. The results are reported in terms of two metrics, descriptions of which 
follow and details of which can be found in the Reading Report Card. 

• Average NAEP reading scale scores are reported for all students or for subgroups 
of students. 

• Percentages of students in each achievement level; that is, the students’ scale 
scores place them into three categories: at or above the Basic level, at or above the 
Proficient level, and at the Advanced level. Also reported are the percentages of 
students whose scores are below the Basic level. 

The bulleted statements in the text as well as the symbols in the tables are based 
on the results of statistical tests of the data. The reader is cautioned to rely on the results 
of these statistical tests rather than on the apparent magnitude of any difference in scale 
scores or percentages in making inferences from the data. 

The NAEP Reading Scale 

Students’ responses to the NAEP 1998 reading assessment were analyzed to determine 
the percentages of students responding correctly to each multiple-choice question or in 
each of several score categories for constructed-response questions (requiring a short or 
long written answer). Three scales were developed, one for each of the three purposes 
for reading: reading for literary experience; reading to gain information ; and reading to 
perform a task (grades 8 and 12 only). The purposes for reading are described fully in 
the companion Reading Report Card and in the Reading Framework for the National 
Assessment of Educational Progress , 2 both available on the Web (see Appendix A). 
Student performance by each of the purposes for reading appears in the percentile data 
tables at the NAEP Web site. The scales summarize results across all three grades. The 
scale for each of the purposes for reading ranges from 0 to 500 and is linked to its 
corresponding scale from 1992 and 1994. An overall composite scale was developed 
by weighting each of the three subscales based on its relative importance in the NAEP 
reading framework. This composite scale is the metric used to present the average scale 
scores and selected percentiles in this and other reports. 
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Progress: 1992-1998. Washington, DC: Author. 
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The Reading Achievement Levels 

In addition to the NAEP reading scale, results are also reported in terms of the reading 
achievement levels authorized by the NAEP legislation 3 and adopted by the National 
Assessment Governing Board. The achievement levels are performance standards based 
on the collective judgments about what students should be expected to know and to do. 
Viewing students’ performance from this perspective provides some insight into the 
adequacy of students’ knowledge and skills and the extent to which they achieved 
expected levels of performance. The Board reviewed and adopted the recommended 
achievement levels derived from the judgments of a broadly representative panel that 
included teachers, education specialists, and members of the general public. 

For each grade tested, the Board has adopted three achievement levels: Basic, 
Proficient, and Advanced. For reporting purposes, the achievement level cut scores for 
each grade represent the boundaries between four ranges on the NAEP reading scale: 
below Basic, Basic, Proficient, and Advanced. The policy definitions of the achievement 
levels are shown below. The text of the descriptions of expected reading performance 
at each achievement level at grades 4 and 8 can be found under the heading What 
Should Students Be Able to Do? later in this section. The cut scores that divide the 
achievement levels can be found in the footnotes of the tables in Section 2. 



Definitions of the achievement levels 

Basic Partial mastery of prerequisite knowledge and skills that are fundamental 
for proficient work at each grade 

Proficient Solid academic performance for each grade assessed. Students reaching 
this level have demonstrated competency over challenging subject matter, 
including subject-matter knowledge, application of such knowledge to real- 
world situations, and analytical skills appropriate to the subject matter. 

Advanced Superior performance 



The NAEP legislation requires that the achievement levels be used on a 
developmental basis until the Commissioner of Education Statistics determines, as the 
result of a congressionally mandated evaluation by one or more nationally recognized 
evaluation organizations, that the achievement levels are “reasonable, valid, and 
informative to the public.” Upon review of the available information, the Commissioner 
of Education Statistics agrees with the National Academy’s recommendation that caution 
needs to be exercised in the use of the current achievement levels, since in the opinion 
of the Academy appropriate validity evidence for the cut scores is lacking; and the 
process has produced unreasonable results.” 4 Therefore, the Commissioner concludes 
that these achievement levels should continue to be considered developmental and 
should continue to be interpreted and used with caution. The Reading Report Card 
contains further information on the developmental status of the achievement levels. The 
Commissioner and the Governing Board believe that the achievement levels are useful 
for reporting trends in the educational achievement of students in the United States. 



3 The National Education Statistics Act of 1994 requires that the National Assessment Governing Board develop 
“appropriate student performance levels” for reporting NAEP results. 

4 Pelligrino, J. W., Jones, L. R., and Mitchell, K. J. (Eds.). (1999). Grading the nation’s report card : Evaluating NAEP 
and transforming the assessment of educational progress (p. 182). Committee on the Evaluation of the National and 
State Assessments of Educational Progress, Board on Testing and Assessment, Commission on Behavioral and Social 
Sciences and Education, National Research Council. Washington, DC: National Academy Press. 
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Reading Scale Score Results for Public School Students 

Figure 1 on the following page shows graphically Montana’s performance in terms of 
average scale scores as compared to that of the West region and the nation. Note that 
only a small portion of the NAEP reading scale, which actually ranges from 0 to 500, 
is represented on the vertical axis. 

Public School Students, Grade 4 

• In 1998, the average reading scale score of students in Montana was 226. This was 
higher than that of students across the nation (215). 

• In Montana, the average scale score of students was not significantly different in 
1998 (226) from that in 1994 (222). However, the average scale score of fourth 
graders across the nation was higher in 1998 (215) than in 1994 (212). 

Public School Students, Grade 8 

• In Montana, the average reading scale score was 270 in 1998, higher than that of 
students across the nation (261). 

The bulleted statements above give only a few of the assessment’s findings. Complete 
notations of significant differences in overall performance for 1994 as compared to 1998 
can be found in Table 1A in Section 2. Note that the changes indicated between two 
particular years show differences in performance between two distinct points in time and 
are not meant to indicate a general trend through the intervening years. Additional 
information can be found in the NAEP 1998 Reading Report Card and on the NAEP 
Web site. 
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How to read Figure 1 

The vertical bars in Figure 1 show reading scores at grade 4 for 1992, 1994, and 
1998, and at grade 8 for 1998, along only a portion of the NAEP 0-500 reading 
scale. The NAEP reading scale is a cross-grade scale (that is, it covers all three 
grades), so the difference between the performance of fourth graders and eighth 
graders can be observed. The reader must be sure to look at the actual scores (on 
top of the bars) rather than at the comparative heights of the bars when comparing 
performance for the two grades. (Table 1A in Section 2 also shows these scores, 
their associated standard errors, and the scores’ distribution at selected percentiles.) 

Montana’s overall average scale score is followed by those for the West region and 
for the nation. The regional and national data come from the national sample, which 
is a different sample from that providing the state data. However, through linking 
procedures, state and national reading results are placed on the same scale. 
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Average reading scale scores for public school students at grades 4 and 8 




Grade 4 Grade 4 Grade 4 Grade 8 



SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1992, 1994, and 1998 Reading Assessments. 
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Achievement Level Results for Public School Students 

Figure 2 on the following page shows graphically Montana’s performance in terms of 
achievement levels (see page 3 for a description) as compared to that of the West region 
and the nation. 



• In 1998, the percentage of Montana students who performed at or above the 
Proficient level was 37 percent. This percentage was greater than the percentage 
of the nation’s students who performed at the same level (29 percent). 

• The percentage of students performing at or above the Proficient level in Montana 
did not differ significantly in 1998 (37 percent) from that in 1994 (35 percent). 



• In 1998, the percentage of students who performed at or above the Proficient level 
in Montana was 38 percent. This percentage was greater than that of students across 
the nation (31 percent). 

The bulleted statements above give only a few of the assessment’s findings. Complete 
notations of significant changes in overall performance for 1994 as compared with 1998 
can be found in Table IB in Section 2. 



Figure 2 shows the percentages of fourth- and eighth-grade students whose scores 
on the NAEP reading assessment placed them into each of the three achievement 
levels, or into the range below the Basic achievement level. In order to place 
students in the achievement level categories, scale score cutpoints were set for each 
level at each grade in 1992, the first year of the assessment. The process of 
determining the achievement levels is under development; thus, they should be used 
and interpreted with caution. The Introduction to the NAEP 1998 Reading Report 
Card contains further information on the developmental status of achievement levels. 

To assist comparisons of performance across the years or comparisons at the state, 
regional, and national levels, the bars are centered between the top of the Basic 
category and the beginning of the Proficient category. There is a vertical line 
representing 0 percent through this location. A visual comparison of the 
percentages of students performing at or above the Proficient level in the jurisdiction 
and the nation can be made by comparing the extension of the bars to the right of 
the zero axis. 



Public School Students, Grade 4 



Public School Students, Grade 8 



How to read Figure 2 
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FIGURE 2 



Reading achievement level results for public school students at grades 4 
and 8 



The bars below contain estimated percentages of students in each NAEP reading achievement category. Each population 
of students is aligned at the point where the Proficient category begins, so that they may be compared at Proficient and above. 
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SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1992, 1994, and 1998 Reading Assessments. 



The text and tables in this report refer to the percentage of students who score “at 
or above Proficient ” and “at or above Basic.” These percentages are cumulative. For 
instance, in Table IB in Section 2, “at or above Proficient ” appears as a single 
percentage. In order to compare the percentage in Figures 2, 4, and 5 with that in 
Table IB, the percentage appearing in the Proficient band in the figures must be added 
to the percentage in the Advanced band to obtain the percentage of students whose scores 
categorize them as “at or above Proficient.” Similarly, the sum of the percentages 
appearing in the Basic , Proficient , and Advanced bands yields the percentage of students 
“at or above Basic 

Figures 2, 4, and 5 allow one to compare performance by the total percentage of 
a given student population whose scores put the students in the broad category “at or 
above Proficient” (that is, simply comparing bar lengths to the right of the zero axis). 
Other interesting comparisons might consider the components of the bar lengths. For 
instance, one student population with 40 percent of its members performing at or above 
Proficient could have 35 percent at Proficient and 5 percent at Advanced. Another 
student population, also with 40 percent of its members performing at or above 
Proficient , might have 25 percent at Proficient and 15 percent at Advanced. In a similar 
manner, on the left side of the zero axis, much can be gained by comparing the 
percentage of students performing at the Basic level with the percentage below the Basic 
level. 
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Comparisons Between Montana and Other Participating 
Jurisdictions 

In 1998, 45 states and other jurisdictions participated in the reading assessment. Of 
those, 43 at grade 4 and 40 at grade 8 met statistical reporting requirements for 
publishing their public school students’ performance on the NAEP reading assessment. 
The maps in Figure 3 show the participating states and indicate their membership in four 
U.S. geographic regions. Note that the Virgin Islands and the domestic and overseas 
Department of Defense Education Activity schools (DoDEA/DDESS and 
DoDEA/DoDDS) do not belong to any of these regions. Reading results for all 
participating states and other jurisdictions are available at the NAEP Web site. 



Average scale scores: How to read Figure 3 

Figure 3 presents maps comparing Montana’s overall 1998 grade 4 and grade 8 
reading scale scores with those of all other participating states and jurisdictions. The 
different shadings are determined by whether or not Montana’s average scale score 
is significantly different (in a statistical sense) from that of each of the other 
participants in the 1998 NAEP state reading assessment. States that did not 
participate in 1998, or that did not meet reporting guidelines, are also represented 
in the maps. 



Achievement levels: How to read Figures 4 and 5 

Figures 4 and 5 permit comparisons of all participants in the NAEP state 
assessment, in terms of percentages of public school students performing at or 
above the Proficient level as well as those performing at the Basic level and below. 
As with Figure 3, the participating jurisdictions are arranged into categories reflecting 
student performance compared to that in Montana. The jurisdictions are grouped 
by whether the percentage of their students with scores at or above the Proficient 
level (including Advanced) was higher than, not significantly different from, or lower 
than the percentage in Montana. Note that the arrangement of the states and other 
jurisdictions within each category is alphabetical; statistical comparisons among the 
jurisdictions in each of the three groups are not included here. 



These figures are available in color at the NAEP Web site. In the companion 
Reading Report Card , there are additional data tables as well as multiple comparison 
charts permitting comparison of each participating jurisdiction with all others. 
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Montana’s 1998 average reading scale score compared to those for other 
participating jurisdictions for public school students at grades 4 and 8 
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SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Reading Assessment. 
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FIGURE 4 

Achievement levels for reading: Comparing the percentage of public school 
students at or above the Proficient level in Montana with those in other 
participating jurisdictions at grade 4 in 1998 



The bars below contain estimated percentages of students in each NAEP reading achievement category. Each population 
of students is aligned at the point where the Proficient category begins, so that they may be compared at Proficient and above. 
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Differences between states and other jurisdictions may be partially explained by other factors not included in this figure. 

SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Reading Assessment. 
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Achievement levels for reading: Comparing the percentage of public school 
students at or above the Proficient level in Montana with those in other 
participating jurisdictions at grade 8 in 1998 



The bars below contain estimated percentages of students in each NAEP reading achievement category. Each population 
of students is aligned at the point where the Proficient category begins, so that they may be compared at Proficient and above. 
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Differences between states and other jurisdictions may be partially explained by other factors not included in this figure. 



SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 Reading Assessment. 
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What Was Assessed? 

The subject area content for each NAEP assessment is developed through a 
congressionally mandated national consensus process directed by the National 
Assessment Governing Board (NAGB). The objectives for each NAEP assessment are 
described in a framework document that delineates the important content and process 
areas to be measured, as well as the types of exercises to be included in the assessment. 

The Reading Framework for the 1992 and 1994 NAEP reading assessments also 
guided the 1998 reading assessment. This framework was developed under the auspices 
of the Council of Chief State School Officers (CCSSO). Reflecting current theories of 
reading, the Reading Framework describes reading as a complex interaction among the 
reader, the text, and the context of the reading situation. According to this interactive 
view of reading, readers use different strategies and processes depending on the type 
of text being read and the purposes for reading it. 

The framework specifies that students be assessed in reading for three different 
purposes: reading for literary experience, reading to gain information, and reading to 
perform a task. The last purpose is not assessed at grade 4. In order to measure 
students’ abilities to read for different purposes, the assessment is composed of three 
different types of texts, each associated with one purpose for reading. 

While responding to text, readers take different approaches in order to understand 
what is being read. The comprehension process typically involves changing stances, or 
orientations toward the text. The framework describes four reading stances: initial 
understanding, developing an interpretation, personal response, and critical stance. 
These stances are not intended to represent a sequential routine of reading abilities nor 
are they considered hierarchical; rather, they describe reading processes that all readers 
use at any level of development. 

The assessment contains reading materials that were drawn from sources 
commonly available to students in and out of school. These authentic materials were 
considered to be representative of the types of reading experiences typically encountered 
by students. Each student in the state assessment was asked to complete two 25-minute 
blocks, each consisting of a reading passage and associated comprehension questions at 
the appropriate grade level. A combination of multiple-choice and constructed-response 
questions was used to assess students’ understanding of the passages. The passages and 
associated items followed a distribution specified by the framework. 



Distribution of items by purpose for reading 
as specified by the Reading Framework 



Purpose 


Grade 4 


Grade 8 


For literary experience 


55% 


40% 


To gain information 


45% 


40% 


To perform a task 


(none) 


20% 
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What Should Students Be Able to Do? 

The following grade-specific achievement level descriptions focus on the interaction of 
the reader, the text, and the context. The achievement level descriptions reflect what 
readers performing at each achievement level should be able to do. The achievement 
levels are cumulative from Basic to Proficient to Advanced. Each level builds on the 
previous level such that knowledge at the Proficient level presumes mastery of the Basic 
level, and knowledge at the Advanced level presumes mastery of both the Basic and 
Proficient levels. 



THE NATION S FIGURE 6A 



Levels of Reading Achievement at Grade 4 

State Assessment 
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BASIC 

LEVEL 

(208) 



Fourth-grade students performing at the Basic level should demonstrate an 
understanding of the overall meaning of what they read. When reading text 
appropriate for fourth graders, they should be able to make relatively obvious 
connections between the text and their own experiences and extend the ideas in 
the text by making simple inferences. 



PROFICIENT 

LEVEL 

(238) 



Fourth-grade students performing at the Proficient level should be able to 
demonstrate an overall understanding of the text, providing inferential as well as 
literal information. When reading text appropriate to fourth grade, they should be 
able to extend the ideas in the text by making inferences, drawing conclusions, and 
making connections to their own experiences. The connection between the text and 
what the student infers should be clear. 



advanced 

LEVEL 

(268) 



Fourth-grade students performing at the Advanced level should be able to 
generalize about topics in the reading selection and demonstrate an awareness of 
how authors compose and use literary devices. When reading text appropriate to 
fourth grade, they should be able to judge text critically and, in general, give 
thorough answers that indicate careful thought. 
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Levels of Reading Achievement at Grade 8 



BASIC 

LEVEL 

( 243 ) 



Eighth-grade students performing at the Basic level should demonstrate a literal 
understanding of what they read and be able to make some interpretations. When 
reading text appropriate to eighth grade, they should be able to identify specific 
aspects of the text that reflect the overall meaning, extend the ideas in the text by 
making simple inferences, recognize and relate interpretations and connections 
among ideas in the text to personal experience, and draw conclusions based on the 
text. 



PROFICIENT 

LEVEL 

( 281 ) 



Eighth-grade students performing at the Proficient level should be able to show an 
overall understanding of the text, including inferential as well as literal information. 
When reading text appropriate to eighth grade, they should be able to extend the 
ideas in the text by making clear inferences from it, by drawing conclusions, and 
by making connections to their own experiences — including other reading 
experiences. Proficient eighth graders should be able to identify some of the 
devices authors use in composing text. 



ADVANCED 

LEVEL 

( 323 ) 



Eighth-grade students performing at the Advanced level should be able to describe 
the more abstract themes and ideas of the overall text. When reading text 
appropriate to eighth grade, they should be able to analyze both meaning and form 
and support their analyses explicitly with examples from the text; they should be 
able to extend text information by relating it to their experiences and to world 
events. At this level, student responses should be thorough, thoughtful, and 
extensive. 
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Who Was Assessed? 

Selection of Schools and Students 

For the NAEP state assessment, participating schools within a given jurisdiction, and 
students in those schools, were selected using probability sampling methods. These 
methods are described in the NAEP 1998 Reading Report Card. 

Montana’s Participation 

The overall participation rate for schools and students must meet guidelines established 
by NCES and NAGB in order for assessment results to be reported publicly. The NAEP 
state assessment in reading was administered in public schools at grade 4 in 1992, in 
public and nonpublic schools at grade 4 in 1994, and in public and nonpublic schools 
at grades 4 and 8 in 1998. Montana participated in the NAEP reading assessments in 
1994 and 1998 at grade 4 and met minimum guidelines for publication of its public 
school results were met in both years. However, Montana failed to meet minimum 
guidelines for reporting nonpublic school results at grade 4. In 1998 at grade 8, Montana 
met the criteria for reporting both public and nonpublic school results. Details on 
participation rates and guidelines for all participating jurisdictions can be found in 
Appendix A of the Reading Report Card. 

Possible Sources of Bias in Reported Results 

Within a certain state sample that meets the guidelines for publication of results, there 
still may exist possible sources of bias of the results due to nonparticipation of selected 
schools or due to nonparticipation of certain student groups. These possible sources of 
bias are indicated by notations and are specified and described in Appendix A of the 
Reading Report Card. Montana’s public schools received one or more notations to 
indicate the possibility of bias. 

Participation by Students with Disabilities or Who Are Learning English 

NAEP endeavors to assess all students selected in the randomized sampling process 
including students with disabilities (SD) as well as students who are beginning to learn 
English and are classified by their schools as limited English proficient (LEP). Although 
the guidelines used to classify students into these two categories vary from state to state, 
NAEP criteria for inclusion standardize the selection of students. The recent Individuals 
with Disabilities Education Act (IDEA) should bring further consistency to classification 
criteria. The percentages of students classified as SD or LEP in all participating states 
and jurisdictions are available at the NAEP Web site in the all-jurisdiction summary data 
tables (SDTs). Participation guidelines specify levels of SD/LEP student 
nonparticipation that put the sample at risk for nonresponse bias; however, no 
jurisdiction failed to meet these guidelines for the 1998 reading assessment. 

NAEP offers certain accommodations for SD or LEP students who need them (for 
example, large print test booklets or extended time), as described in the NAEP 1998 
Reading Report Card and in a special report to follow. However, school personnel make 
the ultimate decision as to whether or not a particular student should take the assessment 
and whether accommodations are needed. The following table shows the percentage 
of students in Montana who were classified as SD or LEP in 1998 and also the 
percentage of those students who were excluded from NAEP at the discretion of school 
personnel. 
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Students in Montana who are classified as 




limited English proficient (LEP) or having disabilities, 1998 


Percentage of students who are : 


School Type 




Grade 4 


Public Nonpublic 


Classified as LEP 


0% 


— 


Excluded from the assessment due to LEP 


0% 


— 


Classified as students with disabilities 


9% 


... 


Excluded from the assessment due to disability 

Grade 8 


4% 




Classified as LEP 


0% 


0% 


Excluded from the assessment due to LEP 


0% 


0% 


Classified as students with disabilities 


10% 


0% 


Excluded from the assessment due to disability 


3% 


0% 



— Nonpublic schools did not participate or minimum participation guidelines were not met. 

SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1998 
Reading Assessment. 



How Are Performance Differences Reported? 

Because the percentages of students and the average reading scale scores presented in 
this report are based on samples— rather than on the entire population of fourth or eighth 
graders in a jurisdiction — the numbers reported are necessarily estimates. As such, they 
are subject to sampling error, a measure of uncertainty reflected in the standard 
error s of the estimate. When the percentages or average scale scores of certain groups 
are compared, it is essential to take the standard error into account rather than to rely 
solely on observed similarities or differences. The comparisons discussed in this report 
are based on statistical tests that consider both the magnitude of the differences between 
the averages or percentages and the standard errors of those statistics. 

The statistical tests determine whether the evidence — based on the data from the 
groups in the sample — is strong enough to conclude that there is an actual difference in 
the averages or percentages for those groups in the population. If the evidence is strong 
(i.e., the difference is statistically significant), the report describes the group averages 
or percentages as being different (e.g., one group performed higher than or lower than 
another group) regardless of whether the sample averages or percentages appear to be 
about the same or not. If the evidence is not sufficiently strong (i.e., the difference is 
not statistically significant), the averages or percentages are described as being not 
significantly different — again, regardless of whether the sample averages or sample 
percentages appear to be about the same or widely discrepant. 

In this report, statements that compare groups or years by using terms such as 
“higher” or “no significant difference” (e.g., “females scored higher than males” or 
“scores in 1998 were not significantly different from 1994”) are based on the results of 
statistical tests. Change over time that is statistically significant is indicated in the 
tables. The reader is cautioned to rely on the results of the statistical tests (as expressed 
in the bulleted text or as indicated in the tables) rather than on the apparent magnitude 
of any difference in scale scores or percentages in making inferences from the data. 

The statistical tests are discussed in greater detail in the NAEP 1998 Reading Report 
Card and the forthcoming NAEP 1998 Technical Report. 




5 Standard errors measure the uncertainty that another sample drawn from the same population could have yielded 
somewhat different results. 
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SECTION 2 



Overall Reading Performance 
and Performance by 
Demographic Characteristics 



Since its inception in 1969, NAEP’s mission has been to collect, analyze, and produce 
valid and reliable information about the academic performance of students in the United 
States in various learning areas. In 1990, the mission of NAEP was expanded to provide 
state-by- state results on academic achievement. To provide reports with each state’s 
data, the computer-generated reporting system was developed; this report was produced 
using that system. 

From 1990 through 1996, NAEP provided state reports with a variety of variables 
chosen for their general interest to most states. Because of new Internet capabilities, 
and with the approval of the state NAEP representatives, the 1998 state reports are 
tailored to provide information of most immediate need to all states. Consequently, 
results are reported here by total population and broken out by major demographic 
variables only. State NAEP results on the Internet provide resources for customized 
reports not possible in the past. 

Reported in this section are the results for student performance overall as well as 
disaggregated by the main demographic variables usually reported by NAEP: 

• Gender 

• Race/ethnicity 

• Highest level of parental education (grade 8 only) 

• Eligibility for the free or reduced-price school lunch program 

• Type of school location (where applicable) 

• Type of school (public compared to nonpublic, where available) 

Each of these variables is reported first by average scale score and selected percentiles 
and then by percentages of students at or above each achievement level. 

The reader is cautioned against making inferences about the performance of 
students in these groups, or about the effectiveness of the National School Lunch 
Program or types of school control, because there are generally many other factors 
involved that are not discussed here and possibly not addressed by NAEP. 
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Students’ Overall! Scale Scores 

Table 1A shows the overall performance of Montana’s public school students, as well 
as the overall performance for the West region and the nation. The first column of 
results gives the average scale score on the NAEP 0-500 reading scale. 

The columns to the right show the score at each of the selected percentiles. This 
arrangement permits a more detailed view of performance along the distribution of 
lowest to highest scores, with the score at each percentile as a demarcation point — for 
each percentile, that percentage of scores falls below the score at that percentile. For 
instance, 75 percent of the student scores fall below the score shown at the 
75th percentile. If, for a particular jurisdiction, the score at the 75th percentile is equal 
to the national average, 25 percent of its students performed better than the national 
average. A jurisdiction might also compare the score at its 25th percentile for the 
current year to that of a previous year to see what change has occurred for students who 
scored in the lower quartile. 

In terms of the average NAEP reading scale score for Montana’s public school 
students, Table 1A shows the following. 

Public School Students , Grade 4 

• In 1998, the average reading scale score for students in Montana was 226. This was 
higher than that of fourth graders in public schools across the nation (215). 

• In Montana, the average scale score of students was not significantly different in 
1998 (226) from that in 1994 (222): However, the average scale score of fourth 
graders across the nation was higher in 1998 (215) than in 1994 (212). 

Public School Students, Grade 8 

• In 1998, the average scale score of students in Montana was 270, higher than that 
of eighth graders in public schools nationwide (261). 



24 



o 

ERLC 



18 



NAEP 1998 READING STATE REPORT 



Montana 



THE N 
REPORT 
CARO 

1998 


lATlON'S 

rcaep 

—pi 




TABLE 1 A 


Average reading scale scores and selected percentiles for public 
school students: 1994 and 1998 


State Assessment 



Average 

scale 

score 



Scale score distribution 



10th 


25th 


50th 


75th 


90th 


percentile 


percentile 


percentile 


percentile 


percentile 



Grade 


4 
























1998 


Montana 


226 | 


: i.7) 


185 l 


(2.8) 


206 ! 


(2.4) 


228 ( 1.2) 


248 1 


(2.2) 


265 l 


(2.4) 




West 


210 | 


: 2.i) 


157 1 


(2.9) 


184 I 


(2.5) 


213 ( 2.5) 


239 l 


(2.4) 


259 I 


(2.3) 




Nation 


215 | 


: o.8)> 


165 l 


( 2.2)> 


192 l 


(1.1)> 


218 ( 0.9) 


242 | 


[ 1-0) 


261 I 


( 1.3) 


1994 


Montana 


222 | 


: 1.4) 


178 i 


(3.3) 


202 l 


( 19) 


225 ( 1.4) 


246 I 


[ 1.0) 


263 I 


( 1.1) 




West 


212 | 


[ 2.2) 


153 I 


(4.0) 


185 l 


(3.0) 


217 ( 2.8) 


242 | 


[ 1.8) 


262 I 


(2.5) 




Nation 


212 | 


: i.i) 


156 | 


: 2,1) 


187 I 


(1.5) 


217 ( 1.2) 


241 | 


[1.2) • 


261 | 


[ 1.5) 


Grade 


8 
























1998 


Montana 


270 \ 


: i.i) 


232 | 


(3.5) 


253 | 


( 1.6) 


273 ( 1.2) 


291 1 


[ 1.5) 


306 1 


[1.3) 




West 


257 | 


: i.8) 


209 ( 


(3.2) 


234 j 


(2.9) 


260 ( 2.1) 


283 1 


( 1.8) 


300 1 


[1.6) 




Nation 


261 ( 


: o.s) 


215 | 


: i.6) 


239 | 


[1.3) 


264 ( 1.1) 


286 | 


(0.8) 


304 1 


( 1.2) 



The NAEP reading scale ranges from 0 to 500. The standard errors of the statistics in the table appear in parentheses. 
If the notation >(<) appears, it signifies that the 1998 value was significantly higher (lower) than the value for 1994 at 
about the 95 percent confidence level. *** Sample size is insufficient to permit a reliable estimate. 

SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1994 and 1998 
Reading Assessments. 
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Overall Results in Terms of Achievement Levels 

Table IB presents the percentages of students who performed below Basic , at or above 
Basic , at or above Proficient , and at Advanced levels. Because the percentages in the 
levels are cumulative from Basic to Proficient to Advanced , they sum to more than 100 
percent. Only the percentage of students at or above Basic (which includes Proficient 
and Advanced) plus the percentage of students below Basic will always sum to 
100 percent. 

Table IB indicates the following in terms of achievement levels attained by 
Montana’s public school students. 

Public School Students Grade 4 

• In 1998, the percentage of Montana students who performed at or above the 
Proficient level was 37 percent. This was greater than the percentage of the nation’s 
public school students who performed at the same level (29 percent). 

• The percentage of students performing at or above the Proficient level in Montana 
did not differ significantly in 1998 (37 percent) from that in 1994 (35 percent). 

• The percentage of students in Montana who performed at or above the Basic level 
in 1998 (73 percent) was greater than that for the nation’s public school students 
(61 percent). 

Public School Students , Grade 8 

• The percentage of students in Montana who performed at or above the Proficient 
level in 1998 was 38 percent. This percentage was greater than that of public school 
students across the nation (31 percent). 

• In 1998, the percentage of students who performed at or above the Basic level in 
Montana was 83 percent. This percentage was greater than that of public school 
students nationwide (72 percent). 
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Percentages of public school students attaining achievement levels: 
1994 and 1998 


State Assessment 



Below Basic 



At or Above 
Basic 



At or Above 
Proficient 



Advanced 



Grade 

1998 


4 

Montana 


. 27 1 


I 1.8) 


73 1 


(1.8) 


37| 


(2.4) 


8 1 


(12) 




West 


45 1 


I 2.5) 


55 1 


(2.5) 


25 1 


(2.2) 


6 1 


(0.8) 




Nation 


39 1 


; i.o) 


61 i 


(1.0) 


29 j 


(0.9) 


6 1 


(0.5) 


1994 


Montana 


31 1 


; 1 . 7 ) 


69 < 


(1.7) 


35 1 


(1.5) 


71 


(0.7) 




West 


41 < 


! 2.2) 


59 l 


(2.2) 


28 1 


(2.0) 


71 


( 0.8) ■ 




Nation 


41 < 


( 1.1) 


59 1 


(1-1) 


28 j 


( 1.2) 


7 1 


(0.7) 


Grade 

1998 


8 

Montana 


17 1 


( 1.5) 


83 l 


( 1-5) 


38 1 


(1.4) 


2 1 


(0.6) 




West 


33 I 


(2.0) 


67 1 


(2.0) 


27 1 


(19) 


21 


(0.2) 




Nation 


28 1 


(0.9) 


72 j 


(0.9) 


31 j 


(0.9) 


21 


(0.4) 



The achievement levels correspond to the following points on the NAEP reading scale at grade 4 (and 8): Basic, 208-237 
(243-280); Proficient, 238-267 (281-322); and Advanced, 268 (323) and above. The standard errors of the statistics in 
the table appear in parentheses. If the notation >(<) appears, it signifies that the 1998 value was significantly higher 
(lower) than the value for 1994 at about the 95 percent confidence level. *** Sample size is insufficient to permit a 
reliable estimate. 

SOURCE: National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 1994 and 1998 
Reading Assessments. 
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